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Abstract 

Factor analysis was applied to analyze the Noyce Scholarship Program Evaluation 
Scholar Survey, and then further hierarchical linear modeling (HLM) was applied to differentiate 
the variance due to program effects from individual scholars’ effects. This survey was 
constructed to determine scholars’ perception of their individual teacher preparation programs, 
their personal experience, and their decision to teach in high needs schools. Explanatory factor 
analysis was used to combine the many items into a few constructs. Based on theories and 
research, group members’ professional opinions and confirmatory factor analysis, the final eight- 
factor model was determined. Two of the eight factors are scholars’ perceptions of the influence 
of scholarship on becoming teachers and becoming high need teachers. While the two factors 
about scholars’ perceptions of scholarships were entered as the outcome variables in HLM 
analyes, other variables were used as explanatory variables at the scholar level and the program 
level. The results show the program effect and some variables at the scholar level and the 
program level had statistically significant relationships with the outcome variables. 




Statistical Analysis 3 



Using Factor Analysis and Hierarchical Linear Modeling to Analyze STEM Majors' Perceptions 
of the Commitment to Teach in High Need Schools Survey 

Survey is one of the most common methods used for measuring perception. As such, 
appropriately and effectively applying measurement principles and statistical analyses to survey 
data is highly consequential. In addition, time and money used to create and carry out surveys is 
a better investment when the greatest amount of information is collected, and when researchers 
can build statistical models which explain respondents’ perceptions. Therefore, this study 
intended to use the Noyce Scholarship Program Evaluation Scholar (NSPES) Survey to 
demonstrate factor analysis and hierarchical linear modeling (HLM), analyzing which variables 
explain scholars’ decisions to become teachers and teach in high need schools. The motivation 
to use NSPES survey is that there is little information about the financial incentives at the teacher 
recruitment phase (Allen, 2005), and what variables might influence potential candidates, 
especially science, technology, engineering, and mathematics (STEM) majors, to become 
teachers and teach in high need setting. Therefore, in order to investigate this issue carefully, we 
need to use sophisticate techniques like factor analysis and HLM rather than the traditional 
survey data analysis. 

Recruitment and retention of quality teachers in high need schools is especially needed in 
the subject areas of math and science. Teacher quality in mathematics and science education and 
student achievement has been consistently correlated in research literature (Jordan, Mendro, & 
Weerasinghe, 1997; National Research Council [NRC], 2000; Sanders & Rivers, 1996). 
Additionally, the No Child Left Behind Act of 2002 mandates that schools staff classrooms with 
highly qualified teachers. However, recent research indicates that students in high need areas are 
much more likely to be taught by a substitute teacher (National Center for Education Statistics 
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[NCES1, 2006), or a teacher with emergency certification (Christensen & Levine, 1998) than a 
highly qualified teacher. To compound the issue, attrition is greatly elevated in high need areas 
in comparison with other types of environments (Darling-Hammond & Sykes, 2003; Ingersoll, 
2002, 2003; Smith & Ingersoll, 2004). 

To address the shortage of qualified teachers, Darling-Hammond (1997) recommended 
that states and districts “provide scholarships to recruit teachers for ‘high-need’ subjects and 
locations” (p. 4). Many state and federal agencies have heralded this recommendation by 
offering scholarships and financial incentives to address the necessities of high need schools, and 
science and mathematics shortages. Many of these programs are complex involving not only 
financial incentives, but also academic support, extensive field experiences, mentoring and 
employment assistance (see Clewell, Darke, Davis-Googe, Forcier & Manes, 2000) while others 
mainly provide scholarships. Evaluation of these programs often focuses on who the programs 
attract to teaching and their retention while investigations of the influence of disaggregated 
elements is less common (see Clewell et al.). 

The outcomes of scholarship, in terms of how scholarships influence recipients and the 
quality of the pool of educators available for high need areas, have not been investigated 
frequently. However, Darling-Hammond and Sykes (2003) reported positive outcomes from 
scholarship/forgivable loan programs designed to increase the number and quality of teachers 
across subject matter areas, which both reached these goals and increased long-term retention. 

Conversely, Bull, Marks, and Salyer (1994) focused specifically on the influence of 
scholarship programs on attracting individuals to science education and found that the 
scholarships used to draw individuals to teaching actually attracted those who were already 
committed to teaching, thereby not actually increasing the supply pool. They suggested that other 
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factors such as a desire to teach the subject matter, work with children and a sense of social 
justice were greater contributors to teachers’ motivation to teach and that the scholarship 
program simply supported those who were already committed to the profession. Therefore, the 
efficacy of scholarships and loan forgiveness programs on recruiting and retaining qualified 
teachers to high need settings may be affected by the reasons participants actually chose to 
accept scholarships or enroll in loan forgiveness programs. Given the mixed results of 
scholarship/loan forgiveness programs, Darling-Hammond and Sykes (2003) suggested that to be 
effective, scholarship programs should focus not only on academic merit, but also qualities such 
as commitment and perseverance. 

In fact, commitment has been shown to have a significant relationship with teacher job 
satisfaction and eventual retention. Day, Elliot and Kington (2005), in an investigation of 
teachers who had been in the profession for 25-35 years, concluded that commitment is “a set of 
personal and professional values that extend well beyond the traditional ideas of caring and 
dedication” (p. 573) and suggested that low commitments to teaching are related to higher levels 
of turnover, lowered job satisfaction and motivation, while environments that support 
commitments to education in the form of collegial connections and administrative support are 
related to retention. While these studies focus on the relationship between commitment and 
retention, it could be extrapolated that notions of commitment are likely present during 
recruitment and can have implications for whom to recruit and which strategies most effectively 
identify those with the highest levels of commitment. However, the role of financial incentives 
on teachers’ commitments has not been investigated in the literature, suggesting a specific area 
of needed research. While the studies described previously focus broadly on education, it can 
also be deduced that if persons enter high need education with low commitments to those 
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specific environments, but strong commitments to teaching in general, they may be more likely 
to leave high need settings for other areas. 

Furthermore, individuals have cited a myriad of intrinsic, extrinsic, and altruistic reasons 
for entering teaching (Brookhart & Freeman, 1992). These include viewing teaching as a 
vocation and as a contribution to society (Guarino et al., 2006). “Highly persistent” teachers have 
emphasized the intrinsic and moral rewards of teaching as factors influencing their persistence to 
teach (Watt & Richardson, 2008). These intrinsic factors are also coupled with extrinsic factors 
that influence a route to teaching. School environment also appears to be a factor influencing 
teachers’ decision to teach. Justice, Greiner, and Andreson 2003, found that current teachers 
indicated their decision to teach was based on ‘administrative and teacher-to-teacher support, a 
reduction in class size (20-23 students per class, and a district implementation of student 
discipline and school security guidelines.’ Therefore, understanding the perceptions of 
individuals entering a teaching career in STEM fields is imperative for reducing attrition, 
especially for high needs schools and school districts where teacher attrition and turnover are 
quite high (Carroll & Sathya, 2008). 

Except the scholars’ characteristics which influence their decision to become a teacher or 
become a high need school teacher, the characteristics of teacher preparation programs which 
provide specific curricula to train these scholars also have impact on their decisions. Teacher 
certification programs provide scholars with initial teaching experiences and the educational 
knowledge base that is necessary for a successful teaching career. The diversity among teacher 
certification programs has been increasing with about a third of new teachers coming from 
alternative certification programs (Feistritzer and Chester 2002). Different methods used within 
the various teacher certification programs can attribute to a being a prepared teacher and having 
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a successful teaching career (Justice, Greiner, Anderson 2003). Field-based experiences in a 
teacher preparation program appear to play a role in the teaching decision process. When 
teachers did not receive field-based experiences they felt insufficiently prepared when beginning 
teacher and indicated frustration in the areas of classroom management, subject matter 
knowledge, and effective teaching techniques (Justice et al., 2003). Therefore, the characteristics 
of the teacher preparation programs are also important to be considered to impact scholars’ 
perceptions. 

Research Question 

As little is known about the influence of financial incentives at the recruitment phase, the 
purpose of this research was to investigate how these perceptions are affected by the teacher 
preparation programs in which these STEM majors received training. A further purpose is to 
examine and identify which individual (and preparation program) variables are related to 
scholars’ perceptions of their commitment to teach and teach in high need schools because of the 
scholarship. By understanding the effects of variables influencing teachers’ perceived 
commitment to become a teacher and teach in high need school, we hoped to identify variables 
that could be used by teacher preparation programs to design better curriculum to train STEM 
majors to teach in high need schools. In addition, the result might help teacher preparation 
programs and governments target for recruitment potential scholars with certain characteristics. 
In other words, the research questions being answered in this study are: 

1) Which variables are related to a scholar’s perceptions of their commitment to teach? 

2) Which variables are related to a scholar’s perceptions of their commitment to teaching at 
high needs schools? 



Methods 
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Sample Description 

Participants were all Noyce scholarship recipients and they were at a variety of points in 
their careers as educators. Of the 555 scholars responding to the survey, 30.8% were still in their 
teacher preparation/certification programs but not yet full-time teachers, 12.6% were still in their 
programs but also teaching full-time, 45.9% were teaching full-time/part-time, 0.9% taught after 
certification and were working in education but no longer teaching, 0.9% taught after being 
certified but were no longing working in education, 7.6% completed their programs but never 
taught, and 1.3% left their programs without completing certification. These scholars received 
their teacher education from 54 programs around the United States. Of the responding recipients, 
34% indicated they were/are in a graduate program leading to a master’s degree, 28% were/are 
attending an undergraduate program, 21% were/are in a post-baccalaureate program, 12% 
were/are attending a teaching credential program, and 5% were/are attending “other” programs. 

The scholars had strong content backgrounds and reported taking several science and 
mathematics classes within the past few years. About 50% of all recipients indicated taking 10 to 
20 STEM courses. The recipients also had strong backgrounds in teaching methods coursework, 
with over half (53%) reported taking four methods courses, with the majority of all participants 
(84%) reported taking anywhere from one to eight methods courses. About half of all scholars 
worked in a wide array of occupations prior to entering the teacher certification program. 
Average participant age was 27 with a median of 24 and a mode of 22 leading to a positively 
skewed distribution by a few older scholars. Fifty-three percent of the respondents decided to 
become STEM teachers after they were at least 23 years old, while 32% decided to become 
STEM teachers between 19-22 years of age. 
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Because of our interests in the effect of the programs on scholars’ perceptions, it is 
important to describe the teacher preparation/certification programs in which these scholars were 
trained to become STEM teachers in high need settings. Teacher preparation/certification 
programs receiving funding through the Noyce Program exhibit a good deal of variety both 
within and across institutions (Author, 2008). For example, the programs represent an array of 
traditional and alternative certification options. Twenty-nine percent of the scholars described 
their teacher preparation/certification programs as undergraduate programs leading to a 
bachelor’s degree, 12% described their programs as a teaching credential without a degree 
awarded, 23% described their programs as post-baccalaureate or graduate programs without a 
master’s degree awarded, and 36% described their programs as graduate programs with a 
master’ s degree awarded. 

As of 2007, 75 teacher preparation/certification institutions awarded the Noyce 
scholarship to prospective science and mathematics teachers, of which 66 (88%) responded to 
the Noyce Scholarship Program Evaluation: Principal Investigator Survey. The survey was 
administered to the primary investigators (PI) for the Noyce Program at each of the participating 
teacher education programs. Of the nine non-responding Pis, seven were in their first or second 
year and subsequently had no data to report in the survey. Fifty-five percent indicated that the 
Noyce Program paid for over 75% of recipients’ tuition. In general, the Pis indicated that Noyce 
funding greatly increased their ability to recruit a variety of students; however, they perceived 
the Noyce funding as having a lesser effect on their students’ perceptions about teaching careers 
and relationships with the surrounding community, school districts, industry and STEM faculty 



at their institutions. 
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Although the implementation of Noyce varies across programs, generally programs use 
some screening criteria to select scholars, such as GPA (98.5% of all programs), personal 
statement (97%), letters of recommendation (91%), upper level undergraduate status in science 
or mathematics major (82%), structured interviews (73%), bachelor’s degree in the candidate’s 
subject area (73%), and previous work experience (69%) were common areas that Pis considered 
when offering the Noyce Scholarship to students. The intent is to award the scholarship to the 
most qualified STEM candidates in terms of content knowledge and commitment to high need 
education. 

To aid in future retention, some awarding programs also provide specific activities to best 
prepare scholarship and stipend recipients to become successful science and mathematics 
teachers in high need settings. Often this involves providing opportunities for interactions with 
children from different cultures such as mentoring or tutoring, coursework related to diverse 
cultures, and specific instruction about practicing in high need schools. Further experience is 
often provided with opportunities to observe/work on a limited basis through practicum, student 
teaching, and supervised actual classroom teaching in the form of an internship, all experienced 
in high need schools. 

Instrument 

The data utilized in this study were gathered from the Noyce Scholarship Program (NSP) 
Evaluation: Scholar Survey administered online during the summer of 2007. Participants were 
asked to respond to a variety of items regarding their perceptions of and experiences with the 
Noyce Program. The survey consisted of six main components: (a) project overview - scholars’ 
program status and the Noyce scholarship money they can receive, (b) program characteristics 
and organization - details about the experiences, opportunities, and requirements of the programs 
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scholars participated in, (c) teaching environment and experience - for those teachers who were 
currently teaching; scholars’ current teaching status, personal feelings, and school and district 
environments, (d) the decision to become a teacher - timing and factors that led scholars to 
pursue a career in education and indication of the role played by the scholarship, (e) background 
and experience - scholars’ academic background and work experience, and (f) overall 
experience - scholars’ opinion of Noyce Program’s effectiveness. The survey consisted of a 
variety of Likert-scale, multiple choice, and open-ended items. Completion of this survey was 
voluntary. 

However, as described in the sample description section, the 555 scholars were at 
different points in their careers at the time of the survey (i.e., In a teacher certification program, 
not yet a full-time teacher; Completed a teacher certification program, but never taught; Did not 
complete a teacher certification program and will not return; In a teacher certification program 
and teaching full-time as part of that program; Teacher full-time or part-time; Taught after being 
certified and now working in education but not as a teacher; Taught after being certified and now 
not working in education). Therefore, we provided different versions of the survey to these 
scholars. Items did not differ across surveys. However, some items were either included or not 
included in surveys depending on the scholars’ career progress (scholar has not taught yet or left 
their teaching position). 

Statistical Techniques 

To produce quantitative models of the Noyce Scholarship Program (NSP) based on 
scholars’ perceptions, many statistical analyses were performed using data from the NSPE 
survey. A multitude of information (with a total of 83 items) can be obtained from the survey. 
However, because of so many items, it is impractical to use individual items to conduct statistical 
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analyses. As a result, factor analysis was proposed for the analysis of these items. Factor analysis 
has long been held as a powerful tool for developing construct validation. The coalescence of 
multiple indicators of a construct into a single factor provides convergent and methodological 
evidence (Feldt & Brennan, 1989). 

Exploratory factor analysis (EFA, performed with SPSS 16.0) and confirmatory factor 
analysis (CFA, performed with AMOS 1 6) were initially used to reduce the number of items into 
a more manageable number of factors for analysis. The 555 cases were randomly divided into 
two groups, where one group would be used for an EFA and the other group would be used for 
CFA. Principal axis factoring, orthogonal varimax rotation, and pairwise deletion were used in 
the EFA analysis. After examining the scree plot and factor loadings of all items, several 
potential factors were considered. Later, CFA was then performed to determine the fit of the 
factor model to the data, and to provide additional data regarding patterns that emerged. SPSS 
was then used to create factor scores for each factor. Finally, based on theories and research, 
Noyce evaluation group members’ professional opinions, and CFA, the final set of factors were 
agreed upon, and factor scores were estimated. A factor score is a continuous variable as 
opposed to original dichotomous or rating-scale variables. Continuous variables can reflect a 
construct more accurately than the use of a dichotomous or rating scale variable (Bode, 1995; 
Garmoran & Behrends, 1987). The properties of these factor scores more accurately meet the 
assumptions of statistical analyses than dichotomous or rating scale variables. As a result, 
obtaining factor scores allows for further statistical analyses. 

After formation of factors from these individual items, factor scores were modeled using 
two-level hierarchical linear modeling (HLM, performed with HLM 6.02, Raudenbush, Bryk, 
Cheong, & Congdon, 2000). FILM, also known as multi-level modeling, is able to differentiate 
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between the variance due to the program effect and the variance due to individual scholar’s 
effect (Raudenbush & Bryk, 2002). 

A key assumption in single level models such as traditional regression is that the 
observations are independent of one another which is often not true where a nested structure 
exists (e.g., scholars within programs). In these situations, units of observations within a program 
tend to be more similar to one another than observations from other programs. In the present 
study, scholars who come from the same teacher preparation programs may be more similar to 
one another than they are to scholars from different teacher programs. Failing to take into 
account this dependency can result in biased statistical results (Raudenbush & Bryk, 2002). 

HLM takes into the nested structure of this data and therefore reduces this potential bias. 

The formulations of HLMs directly follow the derivation from Raudenbush & Bryk, 

( 2002 ), 

The level 1 model: 

Y ij = Poj + P\j x \ij + filjXlij + •••• + Poj X Qij + r ij (!) 

where: 

• T. are outcomes for person i in level two unit j. It is also a function of individual 
characteristics, X .. . 

• P qj are regression coefficients, q- 0, ..., Q, indicate how the outcome is distributed in 

organization j as a function of the measured person characteristics. 

• X y are the predictors at the level- 1 

• r t j are the random effects at the level-1, and it follows A(0,cr 2 ) 

The level 2 model: 

Pea = Y q 0 + Y q \ w ij + r q 2 W 2j + - + Y q s„ W qSq + U cu 



where: 



(2) 
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• P . are level- 1 coefficients 

• y 0 are level-2 coefficients and are also called fixed effects 

• Wjj are predictors at the level-2 

• u are random effects. 

In this study, grand-mean centering was used for adjusting predictors in equations, and 
maximum likelihood estimation was used to estimate parameters. 

Results 

Eight Factors from EFA and CFA 

After running EFA and CFA, the eight-factor solution (Table 1) was chosen as the one 
that appeared to best fit the data and existing theory about STEM teacher preparation after 
discussion within the Noyce evaluation research group. The content validity of items contained 
in the factors was strengthened by the elimination of few items. A total of 49 out of 71 items 
were used in the eight-factor model. Three items are in the “Scholars’ perception of the influence 
of scholarship on becoming teachers” factor, three items are in the “Scholars’ perception of the 
influence of scholarship on becoming high need teachers” factor, 13 items in the “Preparation for 
high needs schools” factor, seven items in the “Path to teaching” factor, five items in the 
“District/school high needs environment” factor, eight items in the “Personal beliefs towards 
teaching” factor, four items in the “School teaching environment” factor, and six items in the 
“Mentoring experience” factor. Detailed information about the standardized regression weights 
from CFA for the items in each factor is presented in Appendix A. For more detailed information 
about the formation of the eight-factor structure, please see Fawrenz & Fiou (2008) and Fiou, 
Kirchhoff, & Fawrenz (2008). 

Table 1 

The Eight Factor Structure for NSPES Survey 
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Factor 


Factor content 


Role of variable 


Factor 1 


Scholars’ perception of the influence of scholarship on 
becoming teachers 


Outcome variable 


Factor 2 


Scholars’ perception of the influence of scholarship on 
becoming high need teachers 


Outcome variable 


Factor 3 


Preparation for high need schools 


Explanatory variable 


Factor 4 


Path to teaching 


Explanatory variable 


Factor 5 


District/school high need environment 


Explanatory variable 


Factor 6 


Personal beliefs towards teaching 


Explanatory variable 


Factor 7 


School teaching environment 


Explanatory variable 


Factor 8 


Mentoring experience 


Explanatory variable 



As for assessing the fitness of each factor, there are dozens of model fit indexes detailed 
in structural equation modeling (SEM) literature. Described next is a minimal set of fit indexes 
that should be reported and interpreted when reporting the results of SEM analyses in order to 
avoid biases. The fit indexes utilized to assess model fit of the eight-factor model are listed in 
Table 2. They include (1) the model chi-square, (2) the Bentler-Bonett normed fit index (NFI; 
Bender & Bonett, 1980), (3) the Bentler comparative fit index (CFI; Bentler, 1990), and (4) the 
Steiger- Lind root mean square error of approximation (RMSEA; Steiger, 1990). 

As a rule of thumb, the chi-square test is too sensitive to sample size, thus it is not a good 
index to evaluate model fitness; however, it can test the difference in fit between a given over- 
identified model and a just-identified version (Kline, 2005). Based on this suggestion, we do not 
provide an in-depth explanation of the chi-square test as an index for model fitness, however, we 
do elaborate on the guidelines for the other three fit indexes. 

For NFI, a value between 0.90 and 0.95 is acceptable, and above 0.95 is good. For CFI, a 
value greater than 0.9 indicates reasonably good fit of the model (Hu & Bentler, 1999). For 
RMSEA, the closer the value is to 0, the better the fit. According to Browne & Cudeck (1993), 

A value of the RMSEA of about .05 or less would indicate a close fit of the 



model in relation to the degrees of freedom. This figure is based on subjective 
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judgment. It cannot be regarded as infallible or correct, but is more reasonable 
than the requirement of exact fit where the RMSEA = 0.0. We are also of the 
opinion that a value of about 0.08 or less for the RMSEA would indicate a 
reasonable error of approximation and would not want to employ a model with a 
RMSEA greater than 0.1. 

Based on the above criteria, we consider that the responses to the 13 items in the 
“Preparation for high need schools” factor, the five items in the “District/school high need 
environment” factor, the eight items in the “Personal beliefs towards teaching” factor, and the six 
items in the “Mentoring experience” factor do not seem to fit one dimension well because these 
items fail to meet standard criteria for all three tests (Table 2). On the other hand, the “Path to 
teaching” factor and “School teaching environment” factor do fit one dimension well as 
demonstrated by their model fit values. As for the “Scholars’ perception of the influence of 
scholarship on becoming teachers” factor and the “Scholars’ perception of the influence of 
scholarship on becoming high need teachers” factor, the fit index cannot be computed. Both of 
factors are composed of three items, so the number of parameters equal to the number of distinct 
sample moments. In other words, there is no degree of freedom for computing fit index. 

Although not all factors have statistical evidence supporting their goodness of fit, Noyce 
evaluation research group members still considered these factors as having good content validity 
according to existing theory. Moreover, Cronbach’s alpha of the factors indicated they have 
median to strong internal consistency. 



Table 2 



Cronbach ’.v Alpha and Fit Indexes for Eight Factors in NSPES Survey 





Cronbach’s 


Chi- 








Factor 


alpha 


square 


D.F. 


NFI 


CFI RMSEA 


Factor 1: Scholars’ perception of the 
influence of scholarship on becoming 


0.885 


N/A 


N/A 


N/A 


N/A N/A 
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Factor 


Cronbach’s 

alpha 


Chi- 

square 


D.F. 


NFI 


CFI 


RMSEA 


teachers 


Factor 2: Scholars’ perception of the 


0.888 


N/A 


N/A 


N/A 


N/A 


N/A 



influence of scholarship on becoming 
high need teachers 



Factor 3: Preparation for high need 
schools 


0.730 


328.653 


65 


0.646 


0.686 


0.121 


Factor 4: Path to teaching 


0.722 


31.047 


14 


0.938 


0.964 


0.066 


Factor 5: District/school high need 
environment 


0.716 


89.400 


5 


0.714 


0.716 


0.247 


Factor 6: Personal beliefs towards 
teaching 


0.611 


55.048 


20 


0.603 


0.659 


0.080 


Factor 7 : School teaching environment 


0.775 


0.769 


2 


0.994 


1 


0.000 


Factor 8: Mentoring experience 


0.724 


106.180 


9 


0.723 


0.732 


0.198 



After forming the eight factors, factor scores created by the multiple regression method 
were assigned for each scholar. Later, HLMs were used to differentiate the program effect from 
the scholars’ factor score variance. HLMs were also used to identity which variables at the 
scholar level and the program level have influence on scholars’ perceptions of their decisions to 
become a teacher or teach in a high needs schools. Therefore, Factor 1: Scholars’ perception of 
the influence of scholarship on becoming teachers, and Factor 2: Scholars’ perception of the 
influence of scholarship on becoming high need teachers were used as the two outcome variables 
in the analysis. Other variables were used as explanatory variables (Table 1). Factor 3 and factor 
8 were mean aggregated within programs and used as level 2 (program) predictors as well as 
level 1 (scholar) predictors. 



Table 3 

Items in Factor 3: Preparation for high need school 

Item content Item option 

Develop specific strategies for teaching 1: None; 2: Touched on it briefly; 3: Spent 

students from diverse racial and ethnic time discussing or doing; 4: Explored in some 

backgrounds depth; 5: Extensive opportunity 

Consider the relationship between education 1: None; 2: Touched on it briefly; 3: Spent 
and social justice and/or democracy time discussing or doing; 4: Explored in some 
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Item content 


Item option 


depth; 5: Extensive opportunity 


Education about how to work in high need 
schools specifically 


1: No; 2: Yes 


Develop specific strategies for teaching 
English language learners (those with limited 
English proficiency) 


1: None; 2: Touched on it briefly; 3: Spent 
time discussing or doing; 4: Explored in some 
depth; 5: Extensive opportunity 


Develop specific strategies for teaching 
students identified with learning disabilities 


1: None; 2: Touched on it briefly; 3: Spent 
time discussing or doing; 4: Explored in some 
depth; 5: Extensive opportunity 


Student teaching experience in a high need 
school 


1: No; 2: Yes 


Supervised actual classroom teaching in high 
need schools (this may be called student 
teaching, internship, etc. in your state) 


1: No; 2: Yes 


Opportunities to observe/work at high need 
schools (not student teaching) 


1: No; 2: Yes 


Education about different cultures 


1: No; 2: Yes 


Student teaching experience 


1: No; 2: Yes 


Opportunities to interact with children from 
different cultures 


1: No; 2: Yes 


Education field experience (e.g. tutoring, 
teacher aide) working in schools with young 
people like those who attend high need schools 


1: No; 2: Yes 


Opportunities to interact with adults from 
different cultures 


1: No; 2: Yes 


Table 4 

Items in Factor 8: Mentoring experience 


Item content 


Item option 


Mentoring experiences provided by your 
certification program during your second year 
of teaching 


1: No; 2: Yes 


Mentoring experiences provided by your 
district during your second year of teaching 


1: No; 2: Yes 


Mentoring experiences provided by your 
certification program during your first year of 
teaching 


1: No; 2: Yes 


Mentoring experiences provided by your 
district during your first year of teaching 


1: No; 2: Yes 


A guaranteed job (assuming successful 
completion of program) at a participating 
school district 


1: No; 2: Yes 


Continuing contact with participants in your 
teacher education program 


1: No; 2: Yes 
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As mentioned in the instrument section, not all of the scholars answered all survey items, 
so not all six explanatory variables can be used in HLMs for all scholars. More detailed 
information about items which were responded to by scholars from certain groups is listed in 
Appendix A. Only Factor 3: Preparation for high need schools, Factor 4: Path to teaching, and 
Factor 8: Mentoring experience can be analyzed in HLMs for the whole scholars. Scholars who 
were currently teaching full- or part-time are the majority in the dataset, and responded to all 
items in the survey. Therefore, these current STEM teachers’ data, a subset of the whole data, 
can be analyzed using all six factors in HLMs. However, because of a concern that small 
numbers of scholars within a program may yield inaccurate estimates, if the number of a group 
of scholars who come from the same program is less than five, these scholars’ data would be not 
used in HLM. Therefore, in the end, 527 scholars from 43 programs were used for analysis. As 
for the current STEM teachers group data, 312 scholars from 42 programs were used for analysis. 
The results of HLMs for the two outcome variables for the two groups (whole scholars group and 
current STEM teachers group) are presented in the following section, respectively. 

HLM Results for Factor 1: Scholars’ perception of the influence of the scholarship on becoming 
teachers 

Two-level hierarchical linear modeling was used to examine the effect of the different 
variables at the scholar level and the variables of the program level on Factor 1: Scholars’ 
perception of the influence of scholarship on becoming teachers. 

A fully unconditional random intercept model was first fit to Factor 1 scores to determine 
how much variation in the outcome variable lies within and between programs. 

For the outcome variable (Factor 1: scholars’ perception of the influence of scholarship 
on becoming teachers) HLM analysis for the whole scholars group, <j 2 = 0.788 is a variance 
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component that provides information about variability within programs, and f 00 = 0.080 is a 
variance component that provides information about variability between programs factor scores 
means. Therefore, the intraclass correlation, p , , is 0.092. It means that 9.2% of Factor 1 
variation is between programs. The chi-square of 97.789 is significant at a = 0.05 and tells us 
that there is significant variation among the program factor score means. Moreover, the average 
reliability, X , of the program factor score means is 0.515. For the current STEM teachers group, 
& 2 = 0.745and f 00 =0.087, so p 1 is 0.105. Therefore, 10.5% of Factor 1 variation is between 

programs. The chi-square of 78.382 is also significant. The average reliability is 0.414. The 
results from both unconditional models suggest that two-level HLMs are needed for both 
analyses of the Factor 1 score. 

We then proceeded to fit conditional models because conditional models adding other 
potential explanatory variables can explain more variability. The two conditional models (1-all 
scholars in equation 3, 2-current STEM teacher group in equation 4) as follows: 



Model 1: HLM for the whole scholar groups (3) 

Level 1 : Whole scholars 

Yy = J3 0j + /(, j ( Factor 3) + j3 2j (Factor A) + J3 3j ( Factor 8) + r (j 
Level 2: Program 

Poj = y (l0 + ;/ () | ( aggregated Factor3 ) + y Q2 (aggregatedFactorS ) + u 0j 

Pu = r w 

Pi j ~ y 20 

P?>j = y 30 

Model 2: HLM for the current STEM teacher groups (4) 

Level 1 : Current STEM teachers 

Yij ~ Poj + fiij(Factor3) + [i 2] ( Factor A) + j3 3j (Factor5 ) + j3 4j (Factor6) + j3 5j (Factorl) + J3 6j (Factor#) 
Level 2: Program 

Poj = y m + y m (aggregatedFactor3) + y l]2 ( agg re gated Factor^ ) + u 0j 
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Pm = Yi o 
/^2 j ~ Y 20 

/^3 j = / 30 
Paj ~ Y 40 

Psj = y 50 

Pbj ~ Y 60 

The results of the two conditional HLMs (for the whole scholar and the current STEM 
teacher group), are presented below in Table 5. 



Table 5 



Impact of Factors on Scholars’ Perception of the Influence of the Scholarship on Becoming 
Teachers for “Whole Scholars” Group and “Current STEM Teachers” Group 





Whole scholars 


Current STEM teachers 




Coef. 


SE 


Coef. 


SE 


Level 2 










Intercept, y m 


0.014 


0.057 


0.015 


0.066 


Aggregated Factor3, ;/ 0l 


-0.213** 


0.102 


-0.270** 


0.124 


Aggregated Factor8, y 02 


0.232 


0.151 


0.256 


0.200 


Level 1 










Factor3, [3 m 


0.054 


0.052 


0.094 


0.055 


Factor4, J3 20 


0.052 


0.049 


0.009 


0.066 


Factor5, [F :{j 


N/A 




0.125 


0.076 


Factor6, /i 40 


N/A 




-0.117 


0.063 


Factor7, ff t) 


N/A 




-0.025 


0.055 


Factor8, fi 60 


0.115** 


0.052 


0.067 


0.075 
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Note. There were 527 scholars in 43 programs in the “whole scholars” group. There were 312 scholars in 42 
programs in the “current STEM teachers” group. Coef. = coefficient. 

**p<.05. *** pc.Ol. 

Data were analyzed to address what kinds of program and scholar variables are 
associated with Factor 1 scores: Scholars’ perception of the influence of the scholarship on 
becoming teachers”. For the whole scholars group, the statistical results show that aggregated 
“Factor 3: Preparation for high need schools” at the level two had a significant impact on Factor 
1 ( J3 = -0.213 ; SE = 0. 102 ; ;>value<0.05). It shows that when the average Factor 3 increased 
one unit above the grand mean Factor 3 scores, the intercept scores decrease 0.213 units after 
controlling for other predictors. Moreover, “Factor 8: Scholar’s mentoring experience” was a 
significant predictor of factor 1 scores ( /? = 0.1 15 ; SE = 0.052 ; 7>value<0.05). This conditional 

model has a 1 = 0.772 and f 00 = 0.067 , so the intraclass correlation, p , , is 0.080. It means that 

this conditional model can explain more variability then the unconditional model by 0.8%. In 
addition, the deviance test for the unconditional and the conditional model is significant 
( Z ; <14.398). 

On the other hand, for the current STEM teachers group, the statistical results also show 
that the average program factor 3 score was a significant predictor of Factor 1 score intercept 
( P = -0.270 ; SE = 0. 124 ; 7>value<0.05). However, for the STEM teachers group, there were no 

variable at the scholar level related to factor 1 scores. This conditional model has a 1 = 0.716 and 
f 00 = 0.072 , so the intraclass correlation, p , , is 0.091. It means that this conditional model also 
can explain more variability then the unconditional model by 1.4%. However, the deviance test 
for the unconditional and the conditional model is not significant ( xl >15.090). Due to the 



parsimony, the conditional model may not be used. 
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HLM Results for Factor 2: Scholars’ perception of the influence of the scholarship on becoming 
high need teachers 

A fully unconditional random intercept model was first fit to the factor score of “Factor 2: 
Scholars’ perception of the influence of the scholarship on becoming high need teachers” to 
estimate variation between programs. 

For the outcome variable (Factor 2: Scholars’ perception of the influence of scholarship 
on becoming high need teachers) HLM analysis for the whole scholars group, a 2 = 0.832 is a 
variance component that provides information about variability within programs, and 
f 00 = 0.042 is a variance component that provides information about variability between 

programs factor scores means. Therefore, the intraclass correlation, p, , is 0.048. It means that 
4.8% of Factor 2 variation is between programs. The chi-square of 66.966 is significant at 
a = 0.05 and tells us that there is significant variation among the program factor score means. 
Moreover, the average reliability, A , of the program factor score means is 0.356. For the current 
STEM teachers group, cr 2 = 0.862andf 00 =0.056, so p [ is 0.061. Therefore, 6.1% of Factor 2 

variation is between programs. The chi-square of 78.382 is also significant. The average 
reliability is 0.295. The results from both unconditional models suggest that two-level HLMs are 
needed for both analyses of the Factor 2 score. 

The results of the conditional HLMs for the two groups (for the whole scholar and the 
current STEM teacher group), are presented in Table 6 below. 

Table 6 

Impact of Factors on Scholars’ Perception of the Influence of Scholarship on Becoming High 

need Teachers for '‘Whole Scholars” Group and “Current STEM Teachers” Group 

Whole scholars Current STEM teachers 
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Coef. 


SE 


Coef. 


SE 


Level 2 










Intercept, ;/ (l(l 


0.025 


0.051 


-0.039 


0.066 


Aggregated Fact or3, y ox 


-0.006 


0.118 


-0.004 


0.160 


Aggregated Factor8, ;/ 02 


-0.078 


0.132 


-0.114 


0.203 


Level 1 










Factor3, 


0.135*** 


0.046 


0.129** 


0.061 


Factor4, /? 20 


-0.005 


0.050 


0.031 


0.063 


Factor5, /L 0 


N/A 




0.078 


0.079 


Factor6, /? 40 


N/A 




-0.040 


0.081 


Factor7, /? 50 


N/A 




-0.098 


0.078 


Factor8, ft H) 


-0.024 


0.055 


0.056 


0.072 



Note. There were 527 scholars in 43 programs in the “whole scholars” group. There were 312 scholars in 42 
programs in the “current STEM teachers” group. Coef. = coefficient. 

**p<. 05. *** pc.01. 



Data were analyzed to address what kinds of program and scholar variables are 
associated with Factor 2 scores: Scholars’ perception of the influence of scholarship on 
becoming high need teachers. For the whole scholars group, the statistical results show that no 
aggregated variable at the level 2 had a significant impact on Factor 2, but “Factor 3: Preparation 
for high need schools” at the level 1 had a significant impact on Factor 2 
( (3 = 0. 135 ; SE = 0.046 ; p-valuecO.Ol). It shows that when a scholar’s Factor 3 score increased 
one unit above the grand mean, the Factor 1 scores of would increase 0.135 units above the mean. 
Further, this conditional model has a 2 = 0.823 and f 00 = 0.038, so the intraclass correlation, p ; , 
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is 0.044. It means that this conditional model can explain more variability then the unconditional 
model by 0.4%. However, the deviance test for the unconditional and the conditional model is 
significant ( x\ >7.412). The conditional model may not be considered to be used. 

On the other hand, for the current STEM teachers group, the statistical results also show 
no aggregated variables at the program level has a significant impact on Factor 1 , but Factor 3 at 
the scholar level also had a significant relationship with Factor 2 {(3 = 0.129 ; SE = 0.061 ; p- 

value<0.05). This conditional model has a 1 - 0.834 and r 00 = 0.061 , so the intraclass 
correlation, p , , is 0.068. It means that this conditional model explain less variability then the 
unconditional model. The conditional model would not be considered to be used. 

Discussion 

This study showed an example of using EFA, CFA and HFMs to analyze survey data. As 
mentioned in the introduction section, survey is one most commonly used tools for measuring 
peoples’ perceptions. When this larger amount of quantitative information is collected, 
researchers may build statistical models which explain respondents’ perceptions. 

In this study, we used the NSPES Survey as the demonstration how these statistical 
methods can be appropriately used in similar situations. NSPES is composed of 7 1 items. A 
multitude of information can be obtained from the survey, but it is undesirable to use individual 
items to do further analysis. First, too many items can be put in analyses. Second, all items from 
the NSPES Survey are either dichotomous or rating-scale, but continuous items can reflect a 
construct more precisely. Therefore, the EFA was utilized to analyze the survey, and further 
CFA was used to confirm the structures. After finalizing the eight-factor structure, factor scores 
were created for the following HFMs analyses. 
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555 scholars from 43 programs responded the NESPES Survey. It is assumed that not 
only scholars’ individual characteristics, but also programs’ characteristics would influence their 
perception about the influence of the scholarship to become teachers and to become high need 
teachers. Especially, scholars coming from the same program tend to have similar characteristics 
because of similar program training and curriculum. Therefore, HLMs were utilized to 
differentiate programs’ effect on scholars’ perceptions and to identity which variables in the 
scholar level and in the program level had influence on scholars’ perceptions. 

The results from HLMs show that program effects can account for some variability of 
scholars’ for outcome variables (Factor 1: Scholars’ perception of the influence of scholarship on 
becoming teachers and Factor 2: Scholars’ perception of the influence of scholarship on 
becoming high need teachers) in both groups (whole scholars group and current STEM teachers 
group). 

For Factor 1 outcome variable, Factor 3: Preparation for high need schools at the program 
level has statistically significant impact on both the whole scholars group and the current STEM 
teachers group. The results show that there was a negative relationship between the aggregated 
factor scores of Factor 3 at the program level and the factor scores of Factor 1 . For Factor 2 
outcome variable, Factor 3 at the scholar level has statistically significant positive impact on 
both groups. However, these explanatory variables are not major influential variables to impact 
the outcome variables, since they cannot explain much variability. 

There were several limitations of this study that require discussion. First, the sample size 
in this study is 555 which is comparatively small for using EFA for exploring the 71 items and 
CFA for validating 49 items. The recommended minimum for a valid factor analysis is 300. 
Moreover, some CFA fit statistics for several factors are not quite satisfactory. We do think the 
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larger sample size would improve this study. However, due to the limited resources, we still had 
to use EFA and CFA on the 555 scholars’ perceptions. 

Second, we analyzed the same data twice for each outcome variables (Factor 1: Scholars’ 
perception of the influence of scholarship on becoming teachers and Factor 2: Scholars’ 
perception of the influence of scholarship on becoming high need teachers), so the issues of 
compounding Type I error have to be considered. 

Third, the data were from a cross-sectional evaluation survey and it did not employ an 
experimental or quasi-experimental design. Therefore, there is no evidence or basis to say causal 
relationships exist among the scholars’ and programs’ characteristics and ultimately their 
perceptions of the commitment to teach and teach in high needs school. The only credible 
statements that can be made from the data are statements regarding correlations. We were also 
unable to track the scholarship recipients and see how their perceptions evolved over time. A 
longitudinal study would have allowed a more thorough understanding of how these perceptions 
evolved into tangible decisions that teachers make when deciding to remain in teaching. 

Another limitation of this study was that this research project was a subset of a program 
evaluation. Because of this, we were limited by the instrument and could not examine variables 
that might have proven to be more relevant to examining the relationship between scholarship 
and perceptions about teaching and recipients’ demographics. As mentioned above, the chosen 
explanatory variables in this study did not have strong impact on reducing variability. Future 
research might use other items that more appropriately measure these constructs and ask about 
decision to teach as well as examining other potentially useful demographic characteristics that 



were unavailable in this study. 
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In summary, the importance of this study lies in the demonstration of how EFA, CFA, 
and HFMs can be used in analyzing survey data. The broader context for the application can be 
applied in similar survey data analyses. 
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Appendix A. 

Eight Factors for the NSPES Survey/Groups of Scholars within the Eight Factors 
Factor 1: Scholars’ perception of the influence of scholarship on becoming teachers 

(the total number of items=3) 

Standardized 



Item 


regression 
weights 
from CFA 


Squared 

multiple 

correlations 


Groups 


IV. 8. a Become a teacher 


0.88 


0.78 


All 


IV.8.b Complete the certification program 


0.86 


0.74 


groups 

All 


IV.8.C Take a teaching job 


0.80 


0.64 


groups 

All 








groups 



Factor 2: Scholars’ perception of the influence of scholarship on becoming high need 
teachers (the total number of items=3) 



Item 


Standardized 
regression 
weights 
from CFA 


Squared 

multiple 

correlations 


Groups 


IV.8.d Teach in a high need school 


0.87 


0.75 


All 

groups 


IV.8.e Remain teaching in a high need school for the 
full term of your commitment 


0.91 


0.83 


All 

groups 


IV.8.f Remain teaching in a high need school beyond 
the full term of your commitment 


0.77 


0.60 


All 

groups 



Factor 3: Preparation for high need schools (the total number of items=13) 



Item 


Standardized 
regression 
weights 
from CFA 


Squared 

multiple 

correlations 


Groups 


II.4.C Develop specific strategies for teaching students 
from diverse racial and ethnic backgrounds 


0.82 


0.68 


All 

groups 


II.4.d Consider the relationship between education and 
social justice and/or democracy 


0.67 


0.45 


All 

groups 


II. 1 .e Education about how to work in high need 
schools specifically 


0.48 


0.23 


All 

groups 


II.4.a Develop specific strategies for teaching English 
language learners (those with limited English 
proficiency) 


0.66 


0.44 


All 

groups 


II.4.b Develop specific strategies for teaching students 
identified with learning disabilities 


0.76 


0.57 


All 

groups 


II. 1 .h Student teaching experience in a high need school 


0.28 


0.08 


All 
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II.2.d Supervised actual classroom teaching in high need 


0.27 


0.08 


groups 

All 


schools (this may be called student teaching, internship, 
etc. in your state) 

II. 1 .f Opportunities to observe/work at high need 


0.31 


0.10 


groups 

All 


schools (not student teaching) 

II. 1 .c Education about different cultures 


0.32 


0.10 


groups 

All 


II. l.g Student teaching experience 


0.18 


0.03 


groups 

All 


II. 1 .b Opportunities to interact with children from 


0.41 


0.17 


groups 

All 


different cultures 

II.2.a Education field experience (e.g. tutoring, teacher 


0.24 


0.06 


groups 

All 


aide) working in schools with young people like those 

who attend high need schools 

II. 1. a Opportunities to interact with adults from 


0.31 


0.10 


groups 

All 


different cultures 






groups 



Factor 4: Path to teaching (the total number of items=7) 






Item 


Standardized 
regression 
weights 
from CFA 


Squared 

multiple 

correlations 


Groups 


IV. 2 What age were you when you began the teacher 
certification program? years 


0.81 


0.65 


All 

groups 


V.3 Did you work full time before becoming a teacher? 
If yes, in what field was the majority of your work? If 
yes, in what field was the majority of your work? 


0.74 


0.54 


All 

groups 


IV. 1 At what point in your life did you decide to 
become a STEM teacher? 


0.74 


0.55 


All 

groups 


V.4 In becoming a teacher, do you consider yourself to 
have made a “career changer”? 


0.72 


0.52 


All 

groups 


V.l.a How many STEM classes were taken? 


0.43 


0.19 


All 

groups 


V.2.a In what year did you last take a formal course for 
college credit in: Mathematics 


0.30 


0.09 


All 

groups 


V.2.b In what year did you last take a formal course for 
college credit in: Science 


0.38 


0.14 


All 

groups 



Factor 5: District/school high need environment (the total number of items=5) 



Item 



Standardized 
regression 
weights 
from CFA 



Squared 

multiple 

correlations 



Groups 



III.4.b Over 33% of teachers lack sufficient training in 
their academic area.(district) 



0.40 



0.16 



4, 5, 6, 7 
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III.3.b Over 33% of teachers lack sufficient training in 
their academic area.(school) 


0.27 


0.07 


4, 5, 6, 7 


III.3.a Over 50% of students receive free or reduced 


0.90 


0.82 


4, 5, 6, 7 


lunch. (school) 

III. 4. a Over 50% of students receive free or reduced 


0.84 


0.71 


4, 5, 6, 7 


lunch. (district) 

III. 1 Which of the following describes your current 
teaching status? 


0.62 


0.38 


4,5 




Factor 6: Personal beliefs towards teaching (the total number of items=8) 




Standardized 








regression 


Squared 






weights 


multiple 




Item 


from CFA 


correlations 


Groups 


III. 2. a I am satisfied with my current teaching job 


0.52 


0.27 


4, 5, 6, 7 


III.2.C If I had to do it all over again, in view of my 
present knowledge, I would become a teacher 


0.58 


0.33 


4, 5, 6, 7 


III.2.d If I had it to do all over again, I would choose the 
same teacher preparation program and/or route into 
teaching 


0.44 


0.19 


4, 5, 6, 7 


III.2.e In the next three years, I am likely to assume a 
leadership position (e.g., lead teacher, department chair, 
official or unofficial mentor) 


0.52 


0.27 


4,5 


IV.3.f I feel that I have a talent for teaching STEM 


0.26 


0.07 


All 








groups 


III. 5 Within the last three years have you held any 
professional educational leadership positions, e.g., lead 


0.32 


0.10 


4,5 


mathematics teacher, science committee chair, etc. 
IV.3.d I like the flexibility and/or autonomy of STEM 


0.27 


0.07 


All 


teaching 






groups 


IV.3.b I like working with young people 


0.23 


0.06 


All 








groups 



Factor 7: School teaching environment (the total number of items=4) 



Item 



Standardized 
regression 
weights 
from CFA 



Squared 

multiple 

correlations 



111.6. b Strong collaborative leadership (e.g., principals 0.82 0.67 

and other leaders provide teachers with opportunities to 

do well; principals and other leaders can be trusted; 
principals and other leaders share your vision of 
successful classroom practice) 

111. 6. a Collegial relationships (e.g., teachers consult on 0.56 0.32 

the quality of student work and make joint decisions 

based on assessment, collaborate to solve classroom 



Groups 



4, 5, 6, 7 



4, 5, 6, 7 
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challenges, observe and discuss each others 

111. 6. d Mentoring and/or induction support (e.g., 0.63 0.40 4, 5, 6, 7 

organized, supported contact with a more experienced 

teacher, help with issues particular to early career 
teaching) 

111.6. C Availability of supplies or material (e.g., 0.57 0.32 4, 5, 6, 7 

textbooks, print resources, instructional materials such 

as lab supplies or math manipulatives, and classroom 

supplies such as paper, pencils, or tape) 



Factor 8: Mentoring experience (the total number of items=6) 






Item 


Standardized 
regression 
weights 
from CFA 


Squared 

multiple 

correlations 


Groups 


II. 1.1 Mentoring experiences provided by your 
certification program during your second year of 
teaching 


0.83 


0.69 


All 

groups 


Il.l.m Mentoring experiences provided by your district 
during your second year of teaching 


0.45 


0.20 


All 

groups 


II. 1 .j Mentoring experiences provided by your 
certification program during your first year of teaching 


0.75 


0.57 


All 

groups 


II. 1 .k Mentoring experiences provided by your district 
during your first year of teaching 


0.38 


0.15 


All 

groups 


II. 1 .i A guaranteed job (assuming successful completion 
of program) at a participating school district 


0.42 


0.18 


All 

groups 


Il.l.n Continuing contact with participants in your 
teacher education program 


0.38 


0.14 


All 

groups 



Groups 

1 . In a teacher certification program, not yet a full-time teacher 

2. Completed a teacher certification program, but never taught 

3 . Did not complete a teacher certification program and will not return 

4. In a teacher certification program and teaching full-time as part of that program 

5 . Teacher full-time or part-time 

6. Taught after being certified and now working in education but not as a teacher 

7. Taught after being certified and now not working in education 




